Reliability measurement without limits

In computational linguistics, a reliability measurement of 0.8 on some statistic such as $\kappa$ is widely thought to guarantee that hand-coded data is fit for purpose, with lower values suspect. We demonstrate that the main use of such data, machine learning, can tolerate data with a low reliability as long as any disagreement among human coders looks like random noise. When it does not, however, data can have a reliability of more than 0.8 and still be unsuitable for use: the disagreement may indicate erroneous patterns that machine-learning can learn, and evaluation against test data that contain these same erroneous patterns may lead us to draw wrong conclusions about our machine-learning algorithms. Furthermore, lower reliability values still held as acceptable by many researchers, between 0.67 and 0.8, may even yield inflated performance figures in some circumstances. Although this is a common sense result, it has implications for how we work that are likely to reach beyond the machine-learning applications we discuss. At the very least, computational linguists should look for any patterns in the disagreement among coders and assess what impact they will have

Inter-Coder Agreement for Computational Linguistics

This article is a survey of methods for measuring agreement among corpus annotators. It exposes the mathematics and underlying assumptions of agreement coefficients, covering Krippendorff's alpha as well as Scott's pi and Cohen's kappa; discusses the use of coefficients in several annotation tasks; and argues that weighted, alpha-like coefficients, traditionally less used than kappa-like measures in computational linguistics, may be more appropriate for many corpus annotation tasks—but that their use makes the interpretation of the value of the coefficient even harder. </jats:p

Analysis of Sensor Coil Implemented in Maxwell-Wien Bridge Circuit for Detecting Ferrous and Non-Ferrous Particles

This paper presents an analysis of a wear debris sensor coil implemented in a Maxwell-Wien Bridge circuit used in detecting ferrous and non-ferrous particles. The sensor coil is designed to have a diameter approximately 15 times larger than the largest particle being detected. To detect particles up to 2 mm in diameter, a coil 30 mm in diameter was constructed. Simulations of the experiment indicate that the available particles should be detectable by the sensor coil. However, experimental results indicate that the Maxwell-Wien Bridge does not detect the available particles. Further simulations are presented accounting for parasitic capacitance of the sensor coil. The results from these simulations show that large parasitic capacitance affects the behavior of the bridge. The available particles were etched into smaller particles more typical of wear debris. It is expected that for a smaller coil, the effects of parasitic capacitance will be reduced

An Automated Meeting Assistant: A Tangible Mixed Reality Interface for the AMIDA Automatic Content Linking Device

We describe our approach to support ongoing meetings with an automated meeting assistant. The system based on the AMIDA Content Linking Device aims at providing relevant documents used in previous meetings for the ongoing meeting based on automatic speech recognition. Once the content linking device finds documents linked to a discussion about a similar subject in a previous meeting, it assumes they may be relevant for the current discussion as well. We believe that the way these documents are offered to the meeting participants is equally important as the way they are found. We developed a mixed reality, projection based user interface that lets the documents appear on the table tops in front of the meeting participants. They can hand them over to others or bring them onto the shared projection screen easily if they consider them relevant. Yet, irrelevant documents don't draw too much attention from the discussion. In this paper we describe the concept and implementation of this user interface and provide some preliminary results

The NITE XML Toolkit Meets the ICSI Meeting Corpus: Import, Annotation, and Browsing

Abstract. The NITE XML Toolkit (NXT) provides library support for working with multimodal language corpora. We describe work in progress to explore its potential for the AMI project by applying it to the ICSI Meeting Corpus. We discuss converting existing data into the NXT data format; using NXT’s query facility to explore the corpus; hand-annotation and automatic indexing; and the integration of data obtained by applying NXT-external processes such as parsers. Finally, we describe use of NXT as a meeting browser itself, and how it can be used to integrate other browser components.

The effectiveness of intrapartum ultrasonography in assessing cervical dilatation, head station and position: A systematic review and meta-analysis

The objective of this review was to assess the effectiveness of intrapartum ultrasonography in measuring cervical dilatation, head station and position. Electronic literature searches were carried out of MEDLINE, CINAHL, and Web of Knowledge, plus manual reference list checks of all relevant articles. All published prospective studies comparing intrapartum ultrasonography with digital VE in the determination of cervical dilatation, head station and position were then evaluated for the success rate and level of agreement between ultrasonography and digital VE. Ultrasonography had higher success rate than digital VE in the determination of fetal head position, with a statistically significant difference in the first stage of labour. Second, although the successful determination of cervical dilatation was in favour of digital VE, the difference was not statistically significant. In addition, there was high agreement between ultrasound and digital VE findings on cervical dilatation. Lastly, a significant but moderate correlation between digital VE and ultrasound methods was found in the assessment of fetal head station. However, no meta-analysis could be done for the fetal head station due to the methodological differences between ultrasound anatomical landmarks and that of digital VE. The findings suggest that ultrasonography is superior to digital VE in the assessment of fetal head position, but has moderate correlation with digital VE in the assessment of head station. It also showed high agreement with digital VE in the assessment of cervical dilatation with no statistically significant difference in terms of success rate

Mini workshop - Real World Engineering Projects: Discovery-based curriculum modules for first-year students

This mini workshop is organized to provide an interactive forum for the introduction a set of six new curriculum modules developed under IEEE's Real World Engineering Projects (RWEP) program. The modules, which are representative of a larger collection of curriculum modules available to the public via an open-access RWEP web portal, are designed for use in the first-year engineering and computer science classroom, and are hands-on, team-based projects that emphasize the societal impact of the work that engineers do. After a brief introduction to the RWEP program and the six showcased curriculum modules, the authors of the modules will present their ideas and demonstrate the laboratory activities associated with their modules in interactive, informal simultaneous sessions. © 2010 IEEE.published_or_final_versionThe 40th ASEE/IEEE Frontiers in Education Conference (FIE) 2010, Arlington, VA., 27-30 October 2010. In Proceedings of 40th FIE, 2010, p. T2A1-T2A

Mini workshop - Real World Engineering Projects: Discovery-based curriculum modules for first-year students

This mini workshop is organized to provide an interactive forum for the introduction a set of six new curriculum modules developed under IEEE's Real World Engineering Projects (RWEP) program. The modules, which are representative of a larger collection of curriculum modules available to the public via an open-access RWEP web portal, are designed for use in the first-year engineering and computer science classroom, and are hands-on, team-based projects that emphasize the societal impact of the work that engineers do. After a brief introduction to the RWEP program and the six showcased curriculum modules, the authors of the modules will present their ideas and demonstrate the laboratory activities associated with their modules in interactive, informal simultaneous sessions. © 2010 IEEE.published_or_final_versionThe 40th ASEE/IEEE Frontiers in Education Conference (FIE) 2010, Arlington, VA., 27-30 October 2010. In Proceedings of 40th FIE, 2010, p. T2A1-T2A